Ground-to-Aerial Image Synthesis with VQ-GAN and Transformers
Project information
- Category: Artificial Intelligence
- Focus: Computer Vision Transformers VQ-GAN Cross-Domain Image generation
- Tech Stack: Pytorch Transformers (Hugging Face) PIL/OpenCV
- Project date: July 2025
- Official Repository
Overview
Cross-view image synthesis is an extreme domain-transfer problem: a ground-level panorama and its corresponding satellite image share the same geographic location but differ completely in viewpoint, scale, and visual structure. No geometric supervision is available — the model must learn the correspondence purely from paired examples.
The system addresses this with a two-stage VQGAN-Transformer pipeline. A pretrained, frozen VQGAN (ImageNet f16) encodes both views into 16×16 grids of discrete codebook indices — 256 tokens per image. A minGPT Transformer then learns to autoregressively predict the 256 satellite tokens conditioned on the 256 ground-level tokens, trained with cross-entropy on 35,191 CVUSA image pairs over 100 epochs:
Training uses token masking scheduling, mixed-precision AMP, label smoothing, and optionally RoPE positional encoding. The best checkpoint reaches an LPIPS of ~0.44 on held-out pairs, with the VQGAN decoder producing geometrically consistent aerial reconstructions from street-level input alone.
Key Elements
Cross-View Learning
Learning the correspondence between ground-level street photos and overhead aerial imagery.
VQ-GAN
$z_q = \text{argmin}_{e_k} \|E(x) - e_k\|$
Vector Quantized GAN to encode images into discrete codebook tokens.
Autoregressive Generation
$P(s) = \prod_i p(s_i \mid s_{<i}, c_{ground})$
GPT Transformer predicts satellite tokens sequentially, conditioned on the ground-level token sequence. Supports RoPE positional encoding for improved spatial generalisation.
Polar Format
Satellite targets are stored as polar-projected images, aligning their radial structure with the geometry of ground-level panoramic photos.
Contacts
Get in touch with me!